Scenic Parallelization and Simulator Stepping by Eric-Vin · Pull Request #422 · BerkeleyLearnVerify/Scenic

Eric-Vin · 2025-11-21T20:52:00Z

Description

Adds parallelization to scene sampling and simulation.

Issue Link

N/A

Checklist

I have tested the changes locally via pytest and/or other means
I have added or updated relevant documentation
I have autoformatted the code with black and isort
I have added test cases (if applicable)

Additional Notes

N/A

Eric-Vin · 2025-11-21T20:52:47Z

Benchmark results for parallelized scene sampling on Macbook with M4 Pro:

Base Performance (`generate`, numWorkers=0):
   3.09 | ('adjacentOpposingPair.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   5.95 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  16.80 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   6.12 | ('city_intersection.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   7.84 | ('enclosed_visible.scenic', {})
  80.35 | ('fully_occluded.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   1.23 | ('fully_visible.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
 248.27 | ('narrowGoalNew.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   6.42 | ('narrowGoalOld.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  13.92 | ('pedestrian_02.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  12.85 | ('vacuum.scenic', {'numToys': 0, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  19.26 | ('vacuum.scenic', {'numToys': 1, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  22.96 | ('vacuum.scenic', {'numToys': 2, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  36.70 | ('vacuum.scenic', {'numToys': 4, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  86.58 | ('vacuum.scenic', {'numToys': 8, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})

Base + Overhead Performance (`generateBatch`, numWorkers=1):
   4.09 | ('adjacentOpposingPair.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   6.86 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  15.14 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   6.53 | ('city_intersection.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   7.97 | ('enclosed_visible.scenic', {})
  81.75 | ('fully_occluded.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   1.74 | ('fully_visible.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
 215.58 | ('narrowGoalNew.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   7.09 | ('narrowGoalOld.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  14.74 | ('pedestrian_02.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  19.33 | ('vacuum.scenic', {'numToys': 0, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  23.52 | ('vacuum.scenic', {'numToys': 1, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  31.18 | ('vacuum.scenic', {'numToys': 2, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  44.50 | ('vacuum.scenic', {'numToys': 4, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  98.84 | ('vacuum.scenic', {'numToys': 8, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})

Batch Performance (`generateBatch`, numWorkers=8):
   2.62 | ('adjacentOpposingPair.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   2.92 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   4.49 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   2.17 | ('city_intersection.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   2.08 | ('enclosed_visible.scenic', {})
  14.17 | ('fully_occluded.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   0.90 | ('fully_visible.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  48.10 | ('narrowGoalNew.scenic', {'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   2.18 | ('narrowGoalOld.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   4.34 | ('pedestrian_02.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
   4.19 | ('vacuum.scenic', {'numToys': 0, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   5.21 | ('vacuum.scenic', {'numToys': 1, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
   6.15 | ('vacuum.scenic', {'numToys': 2, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  10.69 | ('vacuum.scenic', {'numToys': 4, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})
  18.74 | ('vacuum.scenic', {'numToys': 8, 'meshBasePath': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/meshes')})

codecov · 2025-11-21T21:10:03Z

Codecov Report

❌ Patch coverage is 83.62573% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.73%. Comparing base (156b54b) to head (9689729).

Files with missing lines	Patch %	Lines
src/scenic/core/simulators.py	84.91%	35 Missing ⚠️
src/scenic/core/scenarios.py	78.35%	21 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #422      +/-   ##
==========================================
- Coverage   90.01%   89.73%   -0.28%     
==========================================
  Files          54       54              
  Lines       13585    13829     +244     
==========================================
+ Hits        12228    12410     +182     
- Misses       1357     1419      +62

Files with missing lines	Coverage Δ
src/scenic/__init__.py	`100.00% <100.00%> (ø)`
src/scenic/core/utils.py	`77.83% <100.00%> (+0.64%)`	⬆️
src/scenic/syntax/translator.py	`93.67% <100.00%> (+0.09%)`	⬆️
src/scenic/core/scenarios.py	`90.37% <78.35%> (-4.03%)`	⬇️
src/scenic/core/simulators.py	`91.14% <84.91%> (-4.92%)`	⬇️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Eric-Vin · 2025-12-04T23:02:25Z

Preliminary benchmark results for parallelized simulation on Macbook with M4 Pro with Metadrive:

Base Performance (`generate`, numWorkers=0):
 139.95 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
 251.33 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})

Base + Overhead Performance (`generateBatch`, numWorkers=1):
 137.52 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
 140.97 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})

Batch Performance (`generateBatch`, numWorkers=8):
  16.59 | ('badlyParkedCarPullingIn.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})
  17.21 | ('bypassing_03.scenic', {'mode2D': True, 'map': PosixPath('/Users/evin/Projects/Scenic_Maintenance/assets/maps/CARLA/Town05.xodr')})

Eric-Vin added 4 commits November 6, 2024 16:49

Initial simulator stepping implementation.

ef9d735

Added stepped simulation test.

a452067

Scene sampling parallelization prototype.

07e773b

Scene sampling parallelization.

99f4ac0

Eric-Vin marked this pull request as draft November 21, 2025 20:52

Simulation parallelization

cf3c647

Eric-Vin added 2 commits December 24, 2025 20:55

Added documentation for setSeed.

349727f

Added documentation and tests for simulation parallelization.

335b437

Eric-Vin marked this pull request as ready for review December 25, 2025 19:17

Merge branch 'main' into ScenicParallelization

e5048b4

Eric-Vin changed the title ~~[WIP] Scenic Parallelization~~ Scenic Parallelization Dec 25, 2025

Eric-Vin added 2 commits December 25, 2025 14:10

Merge branch 'main' into SimulatorStepping

61e61b8

xMerge branch 'SimulatorStepping' into ScenicParallelization

9689729

Eric-Vin changed the title ~~Scenic Parallelization~~ Scenic Parallelization and Simulator Stepping Dec 26, 2025

Eric-Vin mentioned this pull request Dec 26, 2025

[WIP] Simulator Stepping Functionality #312

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scenic Parallelization and Simulator Stepping#422

Scenic Parallelization and Simulator Stepping#422
Eric-Vin wants to merge 10 commits intomainfrom
ScenicParallelization

Eric-Vin commented Nov 21, 2025

Uh oh!

Eric-Vin commented Nov 21, 2025 •

edited

Loading

Uh oh!

codecov bot commented Nov 21, 2025 •

edited

Loading

Uh oh!

Eric-Vin commented Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Eric-Vin commented Nov 21, 2025

Description

Issue Link

Checklist

Additional Notes

Uh oh!

Eric-Vin commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Eric-Vin commented Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Eric-Vin commented Nov 21, 2025 •

edited

Loading

codecov bot commented Nov 21, 2025 •

edited

Loading